Light field image processing method for depth acquisition

ABSTRACT

Techniques for capturing three-dimensional image data of a scene and processing light field image data obtained by an optical wavefront sensor in 3D imaging applications are provided. The disclosed techniques provide a depth map of an observable scene from light field information about an optical wavefront emanating from the scene, and make use of color filters forming a color mosaic defining a primary color and one or more secondary colors, and color radial transfer functions calibrated to provide object distance information from the spatio-spectrally sampled pixel data.

This application is a U.S. National Phase Application under 35 U.S.C. §371 of International Application No. PCT/CA2018/051554, filed Dec. 5,2018, which claims priority to U.S. Provisional Application No.62/594,718, filed Dec. 5, 2017, the disclosures of each of which areincorporated herein by reference in their entirety.

TECHNICAL FIELD

The general technical field relates to imaging systems and methods, moreparticularly, to a light field imaging device and image processingmethod for depth acquisition and three-dimensional (3D) imaging.

BACKGROUND

Traditional imaging hardware involves the projection of complexthree-dimensional (3D) scenes onto simplified two-dimensional (2D)planes, forgoing dimensionality inherent in the incident light. Thisloss of information is a direct result of the nature of square-lawdetectors, such as charge-coupled devices (CCD) or complementarymetal-oxide-semiconductor (CMOS) sensor arrays, which can only directlymeasure the time-averaged intensity I of the incident light, not itsphase, φ, or wave vector, k, or angular frequency, ω:I˜<E(t)>; where E(t)=E ₀ cos({right arrow over (k)}·{right arrow over(r)}−ωt+φ).  (1)

Working within this constraint, plenoptic cameras are forced to recoverdepth information through either the comparative analysis of multiplesimultaneously acquired images, complicated machine learning and/orreconstruction techniques, or the use of active illuminators andsensors. Plenoptic cameras generally describe a scene through the“plenoptic function” which parameterizes a light field impingent on anobserver or point by:P=P(x,y,λ,t,V _(x) ,V _(y) ,V _(z) ,p),  (2)where the x and y coordinates define a certain image plane at time t,for wavelength λ, and polarization angle p, as witnessed by an observerat location (V_(x), V_(y), V_(z)). While they may be single- ormulti-sensor based systems, current plenoptic cameras can rely, atminimum, solely on the intensity of light detected by any given pixel ofa sensor array. More practically, existing solutions, such asstereovision or microlensing, sacrifice overall image quality and sensorfootprint by employing multiple sensors or sensor segmentation toaccommodate the various fields of view required to discern depth.

Random binary occlusion masks and coded apertures are other existingapproaches that provide single-sensor solutions with minimal impact onpackaging or overall footprint. However, despite advances in compressedsensing and non-linear reconstruction techniques, these solutions remainhindered by the massive image dictionaries and computational expenseinvolved.

Time-of-flight and structured-light based techniques actively illuminatea scene with pulsed, patterned, or modulated continuous-wave infraredlight, and determine depth via the full return-trip travel time orsubtle changes in the illuminated light pattern. While these techniquesdo not suffer from image segmentation, they generally require additionalactive infrared emitters and detectors which both increase powerconsumption as well as overall device footprint. Similarly, thesetechniques tend to be sensitive to interfering signals, specularreflections, and ambient infrared light, thus limiting their viabilityoutdoors.

While the technical limitations and complexities of the various imagecapture hardware described above represent significant barriers to theproliferation of light field capture technology, the complexity ofprocessing light field images remains a significant gating factor. Tobegin, a raw and uncompressed traditional two-dimensional (x, y) imageat modern camera resolutions can be on the order of 50 megabytes insize. An equivalent four-dimensional (x, y, u, v) light field imagewould be orders of magnitude larger, nearing gigabytes. Such data sizepushes the limits of traditional computing and mobile phone systems interms of bus bandwidth and memory transfer rates as well as pure storagespace availability. Therefore, practical light field devices wouldrequire immense and efficient compression to function using modernsystem on chip architectures typical of mobile devices.

Stereovision approaches inherently require complex computational stepsincluding feature matching and rectification before light field scenescan be reconstructed. Microlensing approaches, due to thehardware-induced, down-sampled nature of the image, require intensivenon-linear reconstruction algorithms which scale exponentially incomputational cost with the number of pixels in the image to return animage at the native camera resolution. Time-of-flight and other activeillumination approaches often do not capture an intensity image andrequire a secondary mono or color capture device. The captured depth mapthen needs to be stitched with the intensity image before light fieldscenes can be processed. Infrared camera systems often have very lowresolution relative to the intensity capture device. This hardwarelimitation requires further software-based up-sampling to match thedepth map spatial resolution to the spatial resolution of the intensityimage.

Challenges therefore remain in the development of techniques foracquiring and processing 3D light field images that can combine directhardware compression and efficient computational reconstructionalgorithms.

SUMMARY

The present description generally relates to light field imagingtechniques for depth mapping and other 3D imaging applications.

The present description provides methods for processing light fieldimage data obtained by an optical wavefront sensor in 3D imagingapplications. More particularly, the disclosed method can be used forgenerating or building a 3D or depth image of an observable scene fromlight field information about an optical wavefront emanating from ascene.

In accordance with one aspect, there is provided a method of capturingthree-dimensional image data of a scene, the method comprising:

-   -   a) diffracting an optical wavefront originating from the scene        according to a diffraction grating pattern having a grating        period along a grating axis, to generate a diffracted optical        wavefront;    -   b) detecting the diffracted optical wavefront in a near-field        regime using a pixel array comprising a plurality of        light-sensitive pixels each associated with a color filter,        thereby obtaining pixel data, the color filters forming a color        mosaic defining a primary color and one or more secondary colors        arranged such that different neighboring pixels associated with        the primary color detect different spatial parts of the        diffracted optical wavefront over a full cycle of said        diffracted optical wavefront, the pixel array having a pixel        pitch along the grating axis that is smaller than the grating        period;    -   c) parsing the pixel data according to said primary and        secondary colors into corresponding primary and secondary color        channels;    -   d) determining a main base component and a main modulating        component of an intensity profile of the diffracted optical        wavefront associated with the primary color channel;    -   e) determining a secondary base component and a secondary        modulating component of intensity profiles of the diffracted        optical wavefronts associated with each of the secondary        channels;    -   f) reconstructing a 2D image of the scene using the main and        secondary base components; and    -   g) creating a depth map of the scene using at least one color        radial transfer function calibrated to provide object distance        information from the modulating component of an associated one        of said color channels.

In some implementations of this method, each color radial transferfunction relates the intensity profile of the diffracted opticalwavefront of the associated color channel to a corresponding focal planeof an image capture device used for said detecting step. The objectdistance information may be obtained from a relation such that Object(Z)˜2D Image×CRTF, where z is the distance of a given object in the scenefrom the focal plane of the image capture device, 2D Image is the 2Dimage reconstructed at step f), and CRTF are the radial transferfunctions obtained from a fitted 5D function of polar coordinates r, ϕ,θ from the focal plane, a pixel number n, and an incident wavelength λ.

In some implementations of this method, the determining a secondary basecomponent and a secondary modulating component of step e) comprisedusing the intensity profile of the diffracted optical wavefrontassociated with the primary color channel for neighboring banks of saidpixels to determine if said neighboring banks have a constructive or adestructive interference offset.

In some implementations of this method, the reconstructing a 2D image ofstep f) comprises normalizing the secondary base components using themain base components.

In some implementations of this method, the reconstructing a 2D image ofstep f) comprises:

-   -   using the main modulating component and the associated color        radial transfer function to obtain said object distance        information; and    -   using the secondary modulating components and the associated        color radial transfer functions in view of said object distance        information to compensate for artefacts from the diffraction of        the optical wavefront in said 2D image.

In some implementations of this method, the creating of a depth map ofstep g) comprises creating a coarse depth map using the main modulatingcomponent and the associated color radial transfer function.

In some implementations of this method, the creating a coarse depth mapcomprises:

-   -   obtaining relative phase information for the primary color        channel from the main modulating component associated thereto;        and    -   obtaining absolute phase information for the primary color        channel from a comparison of said relative phase information to        the color radial transfer function associated with said primary        color channel.

The creating of a depth map may also comprise correcting said coarsedepth map using the one or more secondary modulating components and theassociated color radial transfer functions.

In some implementations of this method, correcting said coarse depth mapcomprises:

-   -   obtaining relative phase information for each secondary color        channel from the main modulating component associated thereto;        and    -   obtaining absolute phase information for each secondary color        channel from a comparison of said relative phase information to        the color radial transfer function associated with the        associated secondary color channel.

In some implementations of this method, the color mosaic defines two ofsaid secondary colors.

In some implementations of this method, the primary color channel is agreen color channel, the secondary color channels are a red channel anda blue channel, and said associated color radial transfer functionsrespectively defined a green radial transfer function GRTF, a red radialtransfer function RRTF and a blue radial transfer function BRTF. Thecolor mosaic may be a Bayer pattern. The creating of a depth map of stepg) may comprise:

i. creating a coarse depth map by:

-   -   obtaining relative phase information for the green color channel        from the main modulating component associated thereto; and    -   obtaining absolute phase information for the green color channel        from a comparison of said relative phase information to the        green radial transfer function GRTF; and

ii. correcting said coarse depth map by:

-   -   obtaining relative phase information for the red and blue        channels from the main modulating component associated thereto;        and    -   obtaining absolute phase information for the red and blue        channels from a comparison of said relative phase information to        the red and blue radial transfer functions RRTF and BRTF.

In accordance with another aspect, there is provided a non-transitorycomputer readable storage medium having stored thereon computerexecutable instructions for obtaining three-dimensional image data of ascene from a diffracted optical wavefront originating from the scene anddiffracted according to a diffraction grating pattern having a gratingperiod along a grating axis, the diffracted optical wavefront havingbeen diffracted in a near-field regime using a pixel array comprising aplurality of light-sensitive pixels each associated with a color filter,thereby obtaining pixel data, the color filters forming a color mosaicdefining a primary color and one or more secondary colors arranged suchthat different neighboring pixels associated with the primary colordetect different spatial parts of the diffracted optical wavefront overa full cycle of said diffracted optical wavefront, the pixel arrayhaving a pixel pitch along the grating axis that is smaller than thegrating period, the computer executable instructions, when executed by aprocessor having received the pixel data, cause the processor to performthe following steps:

-   -   a) parsing the pixel data according to said primary and        secondary colors into corresponding primary and secondary color        channels;    -   b) determining a main base component and a main modulating        component of an intensity profile of the diffracted optical        wavefront associated with the primary color channel;    -   c) determining a secondary base component and a secondary        modulating component of intensity profiles of the diffracted        wavefront associated with each of the secondary channels;    -   d) reconstructing a 2D image of the scene using the main and        secondary base components; and    -   e) creating a depth map of the scene using at least one color        radial transfer function calibrated to provide object distance        information from the modulating component of an associated one        of said color channels.

In some implementations, each color radial transfer function relates theintensity profile of the diffracted optical wavefront of the associatedcolor channel to a corresponding focal plane of an image capture deviceused for said detecting step. The object distance information isobtained from a relation such that Object(Z) ˜2D Image×CRTF, where z isthe distance of a given object in the scene from the focal plane of theimage capture device, 2D Image is the 2D image reconstructed at step d),and CRTF are the radial transfer functions. The value of CRTF isobtained from a fitted function of polar coordinates r, ϕ, θ from thefocal plane, a pixel number n, and an incident wavelength λ.

In some implementations, the determining a secondary base component anda secondary modulating component of step c) comprised using theintensity profile of the diffracted optical wavefront associated withthe primary color channel for neighboring banks of said pixels todetermine if said neighboring banks have a constructive or a destructiveinterference offset.

In some implementations, the reconstructing a 2D image of step d)comprises normalizing the secondary base components using the main basecomponent.

In some implementations, the reconstructing a 2D image of step d)comprises:

-   -   using the main modulating component and the associated color        radial transfer function to obtain said object distance        information; and    -   using the secondary modulating components and the associated        color radial transfer functions in view of said object distance        information to compensate for artefacts from the diffraction of        the optical wavefront in said 2D image.

In some implementations, the creating of a depth map of step e)comprises creating a coarse depth map using the main modulatingcomponent and the associated color radial transfer function. Thecreating a coarse depth map may comprise:

-   -   obtaining relative phase information for the primary color        channel from the main modulating component associated thereto;        and    -   obtaining absolute phase information for the primary color        channel from a comparison of said relative phase information to        the color radial transfer function associated with said primary        color channel.

In some implementations, the creating of a depth map comprisescorrecting said coarse depth map using the one or more secondarymodulating components and the associated color radial transfer function.Correcting said coarse depth map may comprise:

-   -   obtaining relative phase information for each secondary color        channel from the main modulating component associated thereto;        and    -   obtaining absolute phase information for each secondary color        channel from a comparison of said relative phase information to        the color radial transfer function associated with the        associated secondary color channel.

In some implementations, the color mosaic defines two of said secondarycolors.

In some implementations, the primary color channel is a green colorchannel, the secondary color channels are a red channel and a bluechannel, and said associated color radial transfer functionsrespectively defined a green radial transfer function GRTF, a red radialtransfer function RRTF and a blue radial transfer function BRTF. Thecolor mosaic may be a Bayer pattern. The creating of a depth map of stepe) may comprise:

i. creating a coarse depth map by:

-   -   obtaining relative phase information for the green color channel        from the main modulating component associated thereto; and    -   obtaining absolute phase information for the green color channel        from a comparison of said relative phase information to the        green radial transfer function; and

ii. correcting said coarse depth map by:

-   -   obtaining relative phase information for the red and blue        channels from the main modulating component associated thereto;        and    -   obtaining absolute phase information the red and blue channels        from a comparison of said relative phase information to the red        and blue radial transfer functions.

In accordance with another aspect, there is provided a method ofcapturing three-dimensional image data of a scene, the methodcomprising:

-   -   a) diffracting an optical wavefront originating from the scene        according to a diffraction grating pattern having a grating        period along a grating axis to generate a diffracted wavefront;    -   b) detecting the diffracted optical wavefront in a near-field        regime using a pixel array comprising a plurality of        light-sensitive pixels each associated with a color filter,        thereby obtaining pixel data, the color filters forming a color        mosaic defining a primary color and one or more secondary        colors;    -   c) parsing the pixel data according to said primary and        secondary colors into corresponding primary and secondary color        channels;    -   d) determining a main base component and a main modulating        component of an intensity profile of the diffracted optical        wavefront associated with the primary color channel;    -   e) determining a secondary base component and a secondary        modulating component of intensity profiles of the diffracted        wavefront associated with each of the secondary channels;    -   f) creating a depth map of the scene using at least one color        radial transfer function calibrated to provide object distance        information from the modulating component of an associated one        of said color channels.

In some implementations, the pixel array has a pixel pitch along thegrating axis that is the same or greater than the grating period saidmethod further comprising a preliminary step of focusing the opticalwavefront originating from the scene using chromatically dependentfocusing optics.

In some implementations, each color radial transfer function relates theintensity profile of the diffracted optical wavefront of the associatedcolor channel to a corresponding focal plane of an image capture deviceused for said detecting step.

In some implementations, the creating of a depth map of step f)comprises creating a coarse depth map using the main modulatingcomponent and the associated color radial transfer function. Creating acoarse depth map may comprise:

-   -   obtaining relative phase information for the primary color        channel from the main modulating component associated thereto;        and    -   obtaining absolute phase information for the primary color        channel from a comparison of said relative phase information to        the color radial transfer function associated with said primary        color channel.

In some implementations, creating of a depth map comprises correctingsaid coarse depth map using the one or more secondary modulatingcomponents and the associated color radial transfer function. Correctingsaid coarse depth map may comprise:

-   -   obtaining relative phase information for each secondary color        channel from the main modulating component associated thereto;        and    -   obtaining absolute phase information for each secondary color        channel from a comparison of said relative phase information to        the color radial transfer function associated with the        associated secondary color channel.

In some implementations, the primary color channel is a green colorchannel, the secondary color channels are a red channel and a bluechannel, and said associated color radial transfer functionsrespectively defined a green radial transfer function GRTF, a red radialtransfer function RRTF and a blue radial transfer function BRTF. Thecolor mosaic may be a Bayer pattern. The creating of a depth map of stepf) may comprise:

i. creating a coarse depth map by:

-   -   obtaining relative phase information for the green color channel        from the main modulating component associated thereto; and    -   obtaining absolute phase information for the green color channel        from a comparison of said relative phase information to the        green radial transfer function; and

ii. correcting said coarse depth map by:

-   -   obtaining relative phase information for the red and blue        channels from the main modulating component associated thereto;        and    -   obtaining absolute phase information for the red and blue        channels from a comparison of said relative phase information to        the red and blue radial transfer functions, respectively, and to        said green radial transfer function.

In accordance with yet another aspect, there is provided anon-transitory computer readable storage medium having stored thereoncomputer executable instructions for obtaining three-dimensional imagedata of a scene from a diffracted optical wavefront originating from thescene and diffracted according to a diffraction grating pattern having agrating period along a grating axis, the diffracted optical wavefronthaving been diffracted in a near-field regime using a pixel arraycomprising a plurality of light-sensitive pixels each associated with acolor filter, thereby obtaining pixel data, the color filters forming acolor mosaic defining a primary color and one or more secondary colors,the computer executable instructions, when executed by a processorhaving received the pixel data, cause the processor to perform thefollowing steps:

-   -   a) parsing the pixel data according to said primary and        secondary colors into corresponding primary and secondary color        channels;    -   b) determining a main base component and a main modulating        component of an intensity profile of the diffracted optical        wavefront associated with the primary color channel;    -   c) determining a secondary base component and a secondary        modulating component of intensity profiles of the diffracted        wavefront associated with each of the secondary channels;    -   d) creating a depth map of the scene using at least one color        radial transfer function calibrated to provide object distance        information from the modulating component of an associated one        of said color channels.

In some implementations, the pixel array has a pixel pitch along thegrating axis that is the same or greater than the grating period saidmethod further comprising a preliminary step of focusing the opticalwavefront originating from the scene using chromatically dependentfocusing optics.

In some implementations, each color radial transfer function relates theintensity profile of the diffracted optical wavefront of the associatedcolor channel to a corresponding focal plane of an image capture deviceused for said detecting step.

In some implementations, the creating of a depth map of step d)comprises creating a coarse depth map using the main modulatingcomponent and the associated color radial transfer function.

In some implementations, creating a coarse depth map comprises:

-   -   obtaining relative phase information for the primary color        channel from the main modulating component associated thereto;        and    -   obtaining absolute phase information for the primary color        channel from a comparison of said relative phase information to        the color radial transfer function associated with said primary        color channel.

In some implementations, the creating of a depth map comprisescorrecting said coarse depth map using the one or more secondarymodulating components and the associated color radial transfer function.Correcting said coarse depth map may comprise:

-   -   obtaining relative phase information for each secondary color        channel from the main modulating component associated thereto;        and    -   obtaining absolute phase information for each secondary color        channel from a comparison of said relative phase information to        the color radial transfer function associated with the        associated secondary color channel.

In some implementations, the primary color channel is a green colorchannel, the secondary color channels are a red channel and a bluechannel, and said associated color radial transfer functionsrespectively defined a green radial transfer function GRTF, a red radialtransfer function RRTF and a blue radial transfer function BRTF. Thecolor mosaic may be a Bayer pattern.

In some implementations, the creating of a depth map of step d)comprises:

i. creating a coarse depth map by:

-   -   obtaining relative phase information for the green color channel        from the main modulating component associated thereto; and    -   obtaining absolute phase information for the green color channel        from a comparison of said relative phase information to the        green radial transfer function; and

ii. correcting said coarse depth map by:

-   -   obtaining relative phase information for the red and blue        channels from the main modulating component associated thereto;        and    -   obtaining absolute phase information for the red and blue        channels from a comparison of said relative phase information to        the red and blue radial transfer functions, respectively, and to        said green radial transfer function.

In accordance with an aspect, there is provided an image processingmethod for depth acquisition. The method allows processing light fieldimage data representing a discretized sampling of a continuous lightfield or wavefront incident from an observable scene. The light fieldimage data forms an array of image data pixels representing a spatialdistribution of the incident wavefront having been diffracted by adiffraction grating assembly and, optionally, subsequently spectrallyfiltered according to a mosaic color pattern mapped to the array ofimage data pixels. The diffraction grating assembly is used to create adiffracted wavefront having an angularly dependent diffractive patternrelated to the curvature and rate of spread of the incident wavefront,which may be caused by a focusing optical element provided in front ofthe diffracting grating assembly. In color applications, the diffractedwavefront can be further spectrally filtered by an underlying colorfilter array into two or more discrete spectral components. Theintensity of this spectrally filtered, diffracted wavefront is finallymeasured by the underlying pixels to provide the light field image data,typically arranged into an image frame of rows and columns of datapixels. This light field image data will generally be a collection ofintensity, spectral, and angular information about the originalwavefront, sampled at the pixel level. The present image processingmethod can process this light field image data to acquire a depth mapand reconstruct a light field image of the observed scene.

In some implementations, the image processing method can include atleast some of the following steps:

-   -   Splitting the image data pixels into a plurality of individual        spectral or color channels, in accordance with mosaic color        pattern. The color channels can include at least one main color        channel and at least one secondary color channel, each color        channel containing a portion of the image data pixels. The main        color channel includes pixels that sample different spatial        parts (i.e., phase components) of the diffracted wavefront over        a full cycle, unlike the secondary color channel. By way of        example, in one possible implementation, the image data pixels        can be split into green, red and blue channels, where the green        channel is doubly sampled compared to the red and blue channels        (Bayer pattern) and act as the main channel while either of or        both the red and blue channels act as a secondary channel.    -   Determining a base component and a modulating component        associated with the main color channel. In general, the light        field image data can be expressed as a modulated function        including a base component and a modulating component. The base        component represents the non-phase-dependent optical wavefront        from which a conventional 2D image of a scene can be obtained,        and the modulating component results from the phase-dependent        perturbation to the incident wavefront created by the        diffraction grating assembly.    -   Determining a base component and a modulating component of the        at least one secondary channel from the those of the main        channel.    -   Reconstructing a full color 2D image of the scene using the main        and secondary base components, using the main base component as        a normalizing basis set.    -   Creating a depth map from the main modulating component, and,        optionally, the secondary modulating component(s).

In some implementations, the modulating components can provide angularor phase information about the incident optical wavefront, from which adepth map can be generated. More particularly, the angular phasecontained in the modulating components and depth can be linked via therate of spread or tilt of the wavefront caused by the lens system of theimage capture device. Therefore, in some implementations, knowledge ofthe focal position of the image capture device is needed for absolutedepth mapping, but not for relative depth mapping.

The wavefront spread information contained in the modulating componentscan be compared to pre-calibrated, sensor-specific chromatic radialtransfer functions (CRTFs) to obtain absolute phase information from therelative phase information provided by the modulating components. Insome implementations, the relative phase information provided by themodulating component can be mapped to a wavefront spread using theCRTFs, this wavefront spread itself corresponding to an object positionrelative to the focal plane of the image capture device. In someimplementations, a coarse depth map can be provided by the mainmodulating component, which coarse depth map can be corrected orenhanced by the secondary modulating component.

In some conventional techniques, depth mapping can be achieved bygenerating a disparity between two different light field views in eachplane, much like a stereo camera. That is, such conventional techniquesdo not give depth naturally, but rather calculate depth light fieldinformation, such as from disparity or parallax between two or moreplanes. In contrast, in some of the present techniques, the angularspread of an object's wavefront in polar coordinates is measured inpolar coordinates, which can directly map angular spread to depthwithout having to generate disparity through light field comparison,though it is not precluded to do so as well. In other words, some of thepresent techniques capture depth information inherently, and then usethis depth information to construct light field images.

It is to be noted that other method and process steps may be performedprior, during or after the above-described steps. The order of one ormore of the steps may also differ, and some of the steps may be omitted,repeated and/or combined, depending on the application.

In accordance with another aspect, there is provided a non-transitorycomputer readable storage medium with a computer program stored thereon,wherein the computer program is operable to process and/or characterizelight field image data, wherein the computer program instructs one ormore processors to perform various steps of the methods disclosedherein.

Other features and advantages of the present description will becomemore apparent upon reading of the following non-restrictive descriptionof specific embodiments thereof, given by way of example only withreference to the appended drawings.

BRIEF DESCRIPTION OF THE DRAWINGS

FIG. 1 is a schematic perspective view of an example of a light fieldcapture device that can be used to obtain light field image data to beprocessed using the present techniques.

FIG. 2 is a schematic partially exploded perspective view of the lightfield capture device of FIG. 1.

FIG. 3 is a schematic side view of another example of a light fieldcapture device in a frontside illumination configuration, which can beused to obtain light field image data to be processed using the presenttechniques.

FIG. 4 is a schematic side view of another example of a light fieldcapture device in a backside illumination configuration, which can beused to obtain light field image data to be processed using the presenttechniques.

FIG. 5 is a schematic partially exploded side view of another example ofa light field capture device that can be used to obtain light fieldimage data to be processed using the present techniques, where thepropagation of a wavefront of light through the device is schematicallydepicted.

FIGS. 6A and 6B are schematic partially transparent top views of twoother examples of a light field capture device that can be used toobtain light field image data to be processed using the presenttechniques, where the grating axis of the diffraction grating is notaligned with either of the pixel axes.

FIG. 7 is a schematic perspective view of another example of a lightfield capture device that can be used to obtain light field image datato be processed using the present techniques, where the diffractinggrating assembly includes two sets of orthogonally oriented diffractinggratings arranged to alternate in both rows and columns to define acheckerboard pattern.

FIG. 8 is a schematic side view of another example of a light fieldcapture device that can be used to obtain light field image data to beprocessed using the present techniques. The light field capture deviceincludes focusing optics disposed in front of the diffraction gratingassembly and spatio-spectrally spreading the optical wavefrontoriginating from the scene prior to it reaching the diffraction gratingassembly.

FIG. 9 is a flow diagram of an image processing method, in accordancewith a possible embodiment.

FIG. 10 is a diagram illustrating the acquisition of data to build thecolor radial transfer functions according to one variant.

FIGS. 11a to 11D illustrate example of the measured pixel voltage oneach pixel shown on FIG. 10.

FIGS. 12A and 12B are exemplary representations of a color radialtransfer function.

FIG. 13 illustrates a technique to obtain the secondary chromatic radialtransfer functions from a knowledge of the uniform intensity of light tomeasure and the secondary color channel modulation as a function ofangle or distance.

FIG. 14 illustrates a technique using the primary radial chromatictransfer function, to acquire a normalized, scene independent, value tonavigate the secondary chromatic radial transfer functions.

DETAILED DESCRIPTION

In the present description, similar features in the drawings have beengiven similar reference numerals, and, to not unduly encumber thefigures, some elements may not be indicated on some figures if they werealready identified in a preceding figure. It should also be understoodthat the elements of the drawings are not necessarily depicted to scale,since emphasis is placed upon clearly illustrating the elements andstructures of the present embodiments.

In the present description, and unless stated otherwise, the terms“connected” and “coupled”, as well as variants and derivatives thereof,refer to any connection or coupling, either direct or indirect, betweentwo or more elements. The connection or coupling between the elementsmay be mechanical, optical, electrical, operational or a combinationthereof. It will also be appreciated that positional descriptors andother like terms indicating the position or orientation of one elementwith respect to another element are used herein for ease and clarity ofdescription and should, unless otherwise indicated, be taken in thecontext of the figures and should not be considered limiting. It will beunderstood that such spatially relative terms are intended to encompassdifferent orientations in use or operation of the present embodiments,in addition to the orientations exemplified in the figures. Moreparticularly, it is to be noted that in the present description, theterms “over” and “under” in specifying the relative spatial relationshipof two elements denote that the two elements can be either in directcontact with each other or separated from each other by one or moreintervening elements. In the present description, the terms “a”, “an”and “one” are defined to mean “at least one”, that is, these terms donot exclude a plural number of items, unless specifically statedotherwise.

The present description generally relates to techniques for capturingthree-dimensional image data of a scene and processing light field imagedata obtained by an optical wavefront sensor in 3D imaging applications.More particularly, the present description discloses techniques forgenerating or building a 3D or depth image or a combined 2D image anddepth map of an observable scene from light field information about anoptical wavefront emanating from the scene.

In the present description, the term “providing” is used broadly andrefers to, but is not limited to, making available for use, acquiring,obtaining, accessing, supplying, receiving, assigning and retrieving. Byway of example, in some implementations, the provision of the lightfield image data to be processed can involve the act of directlyacquiring the light field image data using a light field capture deviceand making available the light field image data thus acquired. However,in other implementations, the provision of the light field image datacan involve the act of retrieving or receiving previously acquired lightfield image data, for example from a database or a storage medium.

In some implementations, the present techniques involve the specificmanipulation and comparison of the chromatic dependence of diffractionby means of one or more diffractive optical elements paired with anappropriate chromatic encoding mechanism, as well as its use in 3Dimaging. In some implementations, the light field image data to beprocessed into a 3D image forms an array of image data pixelsrepresenting a spatio-spectral distribution of a light field afterdiffraction by a diffraction grating and spectral filtering according toa mosaic color pattern.

In some implementations, the techniques disclosed herein are sensitiveto not only the intensity of an optical wavefront originating from anobservable scene, but also the wavelength, through a specificspatio-spectral subsampling of a generated interference or diffractiongrating pattern, allowing for direct measurement of the chromaticdependence of diffraction, the angle of incidence, the phase, and thepolarization of the optical wavefront. Therefore, light field imagingdevices, for example depth cameras, can acquire more information thantraditional cameras, which typically record only light intensity. Theraw image data captured by light field capture devices can be used orprocessed in a variety of ways to provide multiple functions including,but not limited to, 3D depth map extraction, 3D surface reconstruction,image refocusing, and the like. Depending on the application, the lightfield image data of an observable scene can be acquired as one or morestill images or as a video stream. That is, each individual capturedframe generally contains all relevant information to generate anindividual light field scene. These frames, and their subsequent lightfield scenes, can be combined and played sequentially to act like avideo stream.

The present techniques can be used in imaging applications that requireor can benefit from enhanced depth sensing and other 3D imagingcapabilities, for example to allow a user to change the focus, the pointof view and/or the depth of field of a captured image of a scene.Non-limiting typical observable scene examples could include: a persontaking a selfie using their front facing camera on a mobile phone, a carapproaching an intersection with pedestrians crossing and a stop sign, acar trying to park in a tight parking spot, an individual's handsgesturing to interact with a virtual or augmented reality scene, and thelike.

The present techniques can be applied to or implemented in various typesof 3D imaging systems and methods including, without limitation, lightfield imaging applications using plenoptic descriptions, rangingapplications through the comparative analysis of the chromaticdependence of diffraction, and single-sensor single-image depthacquisition applications. Non-exhaustive advantages and benefits ofcertain implementations of the present techniques can include:compatibility with passive sensing modalities that employ less power toperform their functions; compatibility with single-sensor architectureshaving reduced footprint; enablement of depth mapping functions whilepreserving 2D performance; simple and low-cost integration into existingimage sensor hardware and manufacturing processes; compatibility withconventional CMOS and CCD image sensors; and elimination of the need formultiple components, such as dual cameras or cameras equipped withactive lighting systems for depth detection.

In the present description, the terms “light” and “optical” are used torefer to radiation in any appropriate region of the electromagneticspectrum. More particularly, the terms “light” and “optical” are notlimited to visible light, but can also include invisible regions of theelectromagnetic spectrum including, without limitation, the terahertz(THz), infrared (IR) and ultraviolet (UV) spectral bands. In someimplementations, the terms “light” and “optical” can encompasselectromagnetic radiation having a wavelength ranging from about 175nanometers (nm) in the deep ultraviolet to about 300 micrometers (μm) inthe terahertz range, for example from about 400 nm at the blue end ofthe visible spectrum to about 1550 nm at telecommunication wavelengths,or between about 400 nm and about 650 nm to match the spectral range oftypical red-green-blue (RGB) color filters. Those skilled in the artwill understand, however, that these wavelength ranges are provided forillustrative purposes only and that the present techniques may operatebeyond these ranges.

In the present description, the terms “color” and “chromatic”, andvariants and derivatives thereof, are used not only in their usualcontext of human perception of visible electromagnetic radiation (e.g.,red, green and blue), but also, and more broadly, to describe spectralcharacteristics (e.g., diffraction, transmission, reflection,dispersion, absorption) over any appropriate region of theelectromagnetic spectrum. In this context, and unless otherwisespecified, the terms “color” and “chromatic” and their derivatives canbe used interchangeably with the term “spectral” and its derivatives.

The present techniques can be used to process raw light field image datacaptured with various types of light field capture devices. Non-limitingexamples of such devices are described below and illustrated in FIGS. 1to 11. Other non-limiting examples of light field capture devicescapable of acquiring light field image data that can be processed usingthe image processing techniques described herein are disclosed inco-assigned international PCT patent application No. PCT/CA2017/050686,the contents of which are incorporated herein by reference in theirentirety.

According to another aspect of the invention, there is provided acomputer readable memory storing computer executable instructionsthereon that, when executed by a computer, can perform various steps ofthe image processing method disclosed herein.

As used herein, the term “computer readable memory” is intended to referto a non-transitory and tangible computer product that can store andcommunicate executable instructions for the implementation of varioussteps of the image processing method disclosed herein. The computerreadable memory can be any computer data storage device or assembly ofsuch devices including, for example: a temporary storage unit such as arandom-access memory (RAM) or dynamic RAM; a permanent storage such as ahard disk; an optical storage device, such as a CD or DVD (rewritable orwrite once/read only); a flash memory; and/or other non-transitorymemory technologies. A plurality of such storage devices may beprovided, as can be understood by those skilled in the art. The computerreadable memory may be associated with, coupled to or included in acomputer configured to execute instructions stored in the computerreadable memory in connection with various functions associated with thecomputer.

As used herein, the term “computer” refers broadly to any computing orprocessing unit or device including electronic circuitry that cancontrol and execute, at least partly, instructions required to performvarious steps of the image processing method disclosed herein. Thecomputer can be embodied by a general-purpose computer, a centralprocessing unit (CPU), a microprocessor, a microcontroller, a processingcore, or any other processing resource or any combination of suchcomputer or processing resources configured to operate collectively as aprocessing unit. For example, in some implementations, the processingunit implementing the image processing method described herein can be animage signal processor (ISP) or a digital signal processor (DSP) runningon a system on a chip (SoC), a graphics processing unit (GPU), afield-programmable gate array (FPGA), or a combination thereof.

It will be readily understood that in some implementations, all of thesteps of the processing method disclosed herein may be accomplished by asingle processor, whereas in other implementations one or more of thesteps of such a method may be performed on different processors or atdifferent physical locations. Furthermore, while in some implementationsthe processing of the image data may be performed by a same device alsoperforming the acquisition of the light field, for example a phone ortablet, in other implementation the light field image data may betransmitted to a different location or device and processed separately.

In some implementations, there is provided a method of processing orotherwise operating upon light field image data representing adiscretized sampling of a continuous optical wavefront incident from anobservable scene. The light field image data can be provided as an arrayof image data pixels representing a spatial distribution of the opticalwavefront after diffraction by a diffraction grating assembly and,optionally, subsequently spectral filtering according to a mosaic colorpattern mapped to the array of image data pixels.

Referring to FIGS. 1 and 2, there is provided a schematic representationof an example of a light field capture device 20 for capturing raw lightfield or depth image data about an observable scene 22, which may beused in the context of a method of capturing three-dimensional data ofthe scene according to some embodiments. In the present description, theterm “light field capture device” broadly refers to any image capturedevice capable of capturing raw image data representing a light field orwavefront emanating from a scene and containing information about notonly light intensity at the image plane, but also other light fieldparameters such as, for example, the direction from which light raysenter the device and the spectrum of the light field.

The light field capture device 20 includes a diffraction gratingassembly or structure 24 configured to receive an optical wavefront 26originating from the scene 22. The diffraction grating assembly 24 caninclude at least one diffraction grating 28, each of which having agrating axis 30 and a diffraction grating pattern or refractive indexmodulation pattern 32 having a grating period 34 along the grating axis30. In FIGS. 1 and 2, the diffraction grating assembly 24 includes asingle diffraction grating 28, although more than one diffractiongrating can be provided in other devices, as described below. Thediffraction grating 28 is configured to diffract the incoming opticalwavefront 26, thereby generating a diffracted optical wavefront 36.

Referring still to FIGS. 1 and 2, the light field capture device 20 alsoincludes a pixel array 38 comprising a plurality of light-sensitivepixels 40 disposed under the diffraction grating assembly 24 andconfigured to detect the diffracted optical wavefront 36 as the lightfield image data about the scene 22. The light field capture device 20can also include a color filter array 42 disposed over the pixel array38. The color filter array 42 includes a plurality of color filters 44each associated with one of the light-sensitive pixels. The colorfilters are arranged in a mosaic color pattern and each color filterfilters incident light by wavelength to capture color information at arespective location in the color filter array 42. The color filter array42 is configured to spatially and spectrally filter the diffractedwavefront 36 according to the mosaic color pattern prior to detection ofthe diffracted wavefront 36 by the plurality of light-sensitive pixels40. By providing a color filter array to perform a directspatio-chromatic subsampling of the diffracted wavefront generated bythe diffraction grating assembly prior to its detection by the pixelarray, the light field capture device can be sensitive to not only itsspectral content, but also the angle and intensity of an incidentwavefront of light, as explained further below.

Depending on the application or use, the light field capture device canbe implemented using various image sensor architectures and pixel arrayconfigurations. For example, the light field capture device can beimplemented simply by adding or coupling a diffraction grating assemblyon top of an already existing image sensor including a pixel array and acolor filter array. For example, the existing image sensor can be aconventional 2D CMOS or CCD imager. However, in other implementations,the light field capture device can be implemented and integrallypackaged as a separate, dedicated and/or custom-designed deviceincorporating all or most of its components (e.g., diffraction gratingassembly, pixel array, color filter array).

For example, referring to FIGS. 3 and 4, in some implementations, thelight field capture device 20 can include pixel array circuitry 86disposed either between the diffraction grating assembly and the pixelarray, in a frontside illumination configuration (FIG. 3), or under thepixel array 38, in a backside illumination configuration (FIG. 4). Moreparticularly, the diffraction grating assembly 24 can be directly etchedinto overlying silicon layers in the case of a frontside illuminationarchitecture (FIG. 3), or placed directly atop a microlens array 64 anda color filter array 42 in the case of a backside illuminationarchitecture (FIG. 4). The microlens array 64 is disposed over the pixelarray 38 and includes a plurality of microlenses 66. Each microlens 66is optically coupled to a corresponding one of the light-sensitivepixels 40 and is configured to focus the spatial part of the diffractedwavefront 36 incident upon it onto its corresponding light-sensitivepixel 40.

In frontside illumination technology, the pixel array circuitry 86includes an array of metal wiring (e.g., a silicon layer hosting aplurality of metal interconnect layers) connecting the color filters 44to their corresponding light-sensitive pixels 40. Meanwhile, backsideillumination technology provides opportunities for directly sampling thediffracted wavefront 36 produced by diffraction of an optical waveform26 by the diffraction grating assembly 24. As light does not have topass through the array of metal wiring of the pixel array circuitry 86before reaching the pixel array 38, which otherwise would result in aloss of light, more aggressive diffraction grating designs withincreased periodicity can be implemented. Also, the shorter opticalstack configuration, as shown in FIG. 2, can allow for the diffractiongrating assembly 24 to be positioned in much closer proximity to thelight-receiving surface 68 of the pixel array 38, thereby decreasing therisk of higher-order diffractive effects which could cause undesirablecross-talk between pixel banks. Similarly, the decreased pixel size canallow for direct subsampling of the diffraction grating by the existingimaging wells.

More detail regarding the structure, configuration and operation of thecomponents introduced in the preceding paragraphs as well as otherpossible components of the light field capture device will be describedbelow.

Returning to FIGS. 1 and 2, the diffraction grating 28 includes agrating substrate 46 extending over the color filter array 42. Thegrating substrate 46 has a top surface 48, on which is formed theperiodic refractive index modulation pattern 32, and a bottom surface50. The grating substrate 46 is made of a material that is transparent,or sufficiently transparent, in the spectral operating range to permitthe diffracted wavefront 36 to be transmitted therethrough. Non-limitingexamples of such material include silicon oxide (SiOx), polymers,colloidal particles, SU-8 photoresist, and glasses. For example, in someimplementations the diffraction grating 28 can be configured to diffractthe optical wavefront 26 in a waveband ranging from about 400 nm toabout 1550 nm.

Diffraction occurs when a wavefront, whether electromagnetic orotherwise, encounters a physical object or a refractive-indexperturbation. The wavefront tends to bend around the edges of theobject. Should a wavefront encounter multiple objects, whether periodicor otherwise, the corresponding wavelets may interfere some distanceaway from the initial encounter as demonstrated by Young's double slitexperiment. This interference creates a distinct pattern, referred to asa “diffraction grating pattern” or “diffraction pattern” or“interference pattern”, as a function of distance from the originalencounter, which is sensitive to the incidence angle and the spectralcontent of the wavefront, and the general size, shape, and relativespatial relationships of the encountered objects. This interference canbe described through the evolving relative front of each correspondingwavelet, as described by the Huygens-Fresnel principle.

In the present description, the term “diffraction grating”, or simply“grating”, generally refers to a periodic structure having periodicallymodulated optical properties (e.g., a refractive index modulationpattern) that spatially modulates the amplitude and/or the phase of anoptical wavefront incident upon it. The diffraction grating may also bereferred to as a Transmission Diffraction Mask (TDM). A diffractiongrating can include a periodic arrangement of diffracting elements(e.g., alternating ridges and grooves) whose spatial period—the gratingperiod—is nearly equal to or slightly longer than (e.g., up to a fewtimes longer than) the wavelength of light. An optical wavefrontcontaining a range of wavelengths incident on a diffraction gratingwill, upon diffraction, have its amplitude and/or phase modified, and,as a result, a space- and time-dependent diffracted wavefront isproduced. In general, a diffracting grating is spectrally dispersive sothat each wavelength of an input optical wavefront will be outputtedalong a different direction. However, diffraction gratings exhibiting asubstantially achromatic response over an operating spectral range existand can be used in some implementations. For example, in someimplementations, the diffraction grating can be achromatic in thespectral range of interest and be designed for the center wavelength ofthe spectral range of interest. More particularly, in the case of aBayer patterned color filter array, the diffraction grating can beoptimized for the green channel, that is, around a center wavelength ofabout 532 nm. It is to be noted that when the diffraction grating isachromatic, it is the mosaic color pattern of the color filter arraythat provides the chromatic sub-sampling of the diffraction pattern ofthe diffracted wavefront.

Depending on whether the diffracting elements forming the diffractiongrating are transmitting or reflective, the diffraction grating will bereferred to as a “transmission grating” or a “reflection grating”. Thediffracting gratings described herein are transmission gratings,although the use of reflection gratings is not excluded a priori.Diffraction gratings can also be classified as “amplitude gratings” or“phase gratings”, depending on the nature of diffracting elements. Inamplitude gratings, the perturbations to the initial wavefront caused bythe grating are the result of a direct amplitude modulation, while inphase gratings, these perturbations are the result of a specificmodulation of the relative group-velocity of light caused by a periodicvariation of the refractive index of the grating material. Thediffracting gratings described herein are phase gratings, althoughamplitude gratings can also be used in other implementations.

In FIGS. 1 and 2, the diffraction grating 28 is a phase grating, morespecifically a binary phase grating for which the refractive indexmodulation pattern 32 includes a series of ridges 52 periodicallyspaced-apart at the grating period 34, interleaved with a series ofgrooves 54 also periodically spaced-apart at the grating period 34. Thespatial profile of the refractive index modulation pattern 32 thusexhibits a two-level step function, or square-wave function, for whichthe grating period 34 corresponds to the sum of the width, along thegrating axis 30, of one ridge 52 and one adjacent groove 54. The gratingperiod 34 can range from about 1 μm to about 20 μm, although othervalues are possible. In FIGS. 1 and 2, the grooves 54 are empty (i.e.,they are filled with air), but they could alternatively be filled with amaterial having a refractive index different from that of the ridgematerial. Also, depending on the application, the diffraction grating 28can have a duty cycle substantially equal to or different from 50%, theduty cycle being defined as the ratio of the ridge width to the gratingperiod 34. Another parameter of the diffraction grating 28 is the stepheight 56, that is, the difference in level between the ridges 52 andthe grooves 54. For example, the step height 56 can range from about 0.2μm to about 1 μm. In some scenarios, the step height 56 can be selectedso that the diffraction grating 28 causes a predetermined optical pathdifference between adjacent ridges 52 and grooves 54. For example, thestep height 56 can be controlled to provide, at a given wavelength andangle of incidence of the optical wavefront (e.g. its centerwavelength), a half-wave optical path difference between the ridges andthe grooves. Of course, other optical path difference values can be usedin other applications.

It is to be noted that while the diffraction grating 28 in FIGS. 1 and 2is a linear, or one-dimensional, binary phase grating consisting ofalternating sets of parallel ridges 52 and grooves 54 forming asquare-wave refractive index modulation pattern 32, other light fieldcapture device can employ different types of diffraction gratings, suchas: diffraction gratings where at least one among the grating period,the duty cycle and the step height is variable; diffraction gratingswith non-straight features perpendicular to the grating axis;diffraction gratings having more elaborate refractive index profiles; 2Ddiffraction gratings; and the like.

Referring still to FIGS. 1 and 2, the pixel array 38 includes aplurality of light-sensitive pixels 40 disposed under the color filterarray 42, which is itself disposed under the diffraction gratingassembly 24. In the present techniques, the electromagnetic radiationthat is detected by the light-sensitive pixels 40 as light field imagedata corresponds to an optical wavefront 26 incident from the scene 22,which has been diffracted and spatio-chromatically filtered prior toreaching the pixel array 38. The pixel array 38 can be embodied by aCMOS or a CCD image sensor, but other types of photodetector arrayscould alternatively be used. As mentioned above, the pixel array 38 canbe configured to detect electromagnetic radiation in any appropriateregion of the spectrum. Each light-sensitive pixel 40 of the pixel array38 can convert the spatial part of the diffracted wavefront 36 incidentupon it into accumulated charge, the amount of which is proportional tothe amount of light collected and registered by the pixel 40. Eachlight-sensitive pixel 40 can include a light-sensitive surface andassociated pixel circuitry for processing signals at the pixel level andcommunicating with other electronics, such as a readout unit.

Referring still to FIGS. 1 and 2, the light-sensitive pixels 40 can bearranged into a rectangular grid of rows and columns defined by twoorthogonal pixel axes 58, 60. The number of rows and columns defines theresolution of the pixel array 38. For example, in some implementations,the pixel array 38 can have a resolution of at least 16 pixels, althougha wide range of other resolution values, including up to 40 megapixelsor more, can be used in other applications. It is to be noted that whilethe light-sensitive pixels 40 are organized into a 2D array in FIGS. 1and 2, they may alternatively be configured as a linear array in otherapplications.

The pixel array 38 can also be characterized by a pixel pitch 62. In thepresent description, the term “pixel pitch” generally refers to thespacing between the individual pixels 40 and is typically defined as thecenter-to-center distance between adjacent pixels 40. Depending on thephysical arrangement of the pixel array 38, the pixel pitch 62 along thetwo orthogonal pixel axes 58, 60 may or may not be the same. It is to benoted that a pixel pitch can also be defined along an arbitrary axis,for example along a diagonal axis oriented at 45° with respect to thetwo orthogonal pixel axes 58, 60. It is also to be noted that, in thepresent techniques, a relevant pixel pitch 62 is the one along thegrating axis 30 of the overlying diffraction grating 28, as depicted inFIGS. 1 and 2. As described in greater detail below, in some embodimentsthe grating period 34 of the diffraction grating 28 is selected to belarger than the pixel pitch 62 of the pixel array 38 along the gratingaxis 30, or, said differently, the pixel pitch 62 along the grating axis30 is smaller than the grating period 34. For example, in someimplementations the pixel pitch 62 along the grating axis 30 can rangefrom 1 μm or less to 10 μm, although different pixel pitch values can beused in other implementations.

In the present description, the term “pixel data” refers to the imageinformation captured by each individual pixel and can include intensitydata indicative of the total amount of optical energy absorbed by eachindividual pixel over an integration period. Combining the pixel datafrom all the pixels 40 yields “raw” light field image data about thescene 22. In the present techniques, because the optical wavefront 26incident from the scene 22 is diffracted and spatially and spectrallyfiltered prior to detection, the light field image data providesinformation about not only the intensity of the incident wavefront 26,but also other light field parameters such as its angle of incidence,phase and spectral content. More particularly, it will be understoodthat the present techniques can allow recovery or extraction of depth orother light field information from the intensity-based diffractionpattern captured by the pixel array 38, as described further below.

Referring still to FIGS. 1 and 2, the color filter array 42 is spatiallyregistered with the pixel array 38, such that each color filter 44 isoptically coupled to a corresponding one of the light-sensitive pixels40. That is, each color filter 44 covers a single light-sensitive pixel40, such that there is a one-to-one relationship, or mapping, betweenthe color filters 44 and the light-sensitive pixels 40. However, inother implementations, each color filter can be optically coupled to atleast two corresponding ones of the plurality of light-sensitive pixels.In both configurations, however, the color filter array 42 and the pixelarray 38 together enable the direct spatio-chromatic sampling of thediffracted wavefront produced by the overlying diffraction gratingassembly 24, as detailed and explained below.

In FIGS. 1 and 2, the color filters 44 are physically organizedaccording to a mosaic color pattern or configuration. The color filtersform a color mosaic defining a primary color and one or more secondarycolors arranged such that different neighboring pixels associated withthe primary color detect different spatial parts of the diffractedoptical wavefront over a full cycle of the diffracted optical wavefront.In some implementations, each color filter 44 is one of a red passfilter, a green pass filter and a blue pass filter. For example, inFIGS. 1 and 2, the mosaic color pattern of the color filter array 42 isa Bayer pattern, in which the color filters arranged in a checkerboardpattern with rows of alternating red (R) and green (G) filters areinterleaved with rows of alternating green (G) and blue (B) filters. ABayer pattern contains twice as many green filters as red or bluefilters, such that the green component of the mosaic color pattern canbe referred to as the primary of main color or as a “densely sampled”component. and both the red and blue components can be referred to assecondary colors or “sparsely sampled” components. In otherapplications, the mosaic color pattern can be embodied by more elaborateBayer-type patterns, for example Bayer-type patterns with an n-pixelunit cell, where n is an integer greater than 4. Of course, the presenttechniques are not limited to Bayer-type patterns, but can be applied toany appropriate mosaic color pattern including, but not limited to, RGB,RGB-IR, RGB-W, CYGM, CYYM, RGBE, RGBW #1, RGBW #2, RGBW #3, andmonochrome. It is to be noted that in some implementations, the colorfilter array 42 may be extended beyond the standard visible Bayerpattern to include hyperspectral imaging and filtering techniques orinterferometric filtering techniques. In such cases, the design of thediffraction grating 28 (e.g., the grating period 34) can be adjusted toaccommodate the increased spectral sampling range.

Referring now to FIG. 5, there is shown a schematic partially explodedside view of a light field capture device 20 suitable for monochromeimaging applications. The light field capture device 20 sharessimilarities with the one shown in FIGS. 1 and 2, in that it includes adiffraction grating 28 disposed on top of a pixel array 38 oflight-sensitive pixels 40 and associated color filter array 42. Thediffraction grating 28 is a binary phase transmission grating having aduty cycle of 50% and a periodic refractive index modulation pattern 32consisting of alternating sets of ridges 52 and grooves 54. FIG. 5 alsodepicts schematically the propagation of light through the device 20. Inoperation, the light field capture device 20 has a field of viewencompassing an observable scene 22. The diffraction grating 28 receivesan optical wavefront 26 (solid line) incident from the scene 22 on itsinput side, and diffracts the optical wavefront 26 to generate, on itsoutput side, a diffracted optical wavefront 36 (solid line) thatpropagates through the color filter array 42 toward the pixel array 38for detection thereby. For simplicity, the incoming optical wavefront 26in FIG. 5 corresponds to the wavefront of a plane wave impinging on thediffraction grating 28 at normal incidence. However, the presenttechniques can be implemented for an optical wavefront of arbitraryshape incident on the diffraction grating 28 at an arbitrary anglewithin the field of view of the light field capture device.

Referring still to FIG. 5, the diffracted wavefront 36 can becharacterized by a diffraction pattern whose form is a function of thegeometry of the diffraction grating 28, the wavelength and angle ofincidence of the optical wavefront 26, and the position of theobservation plane, which corresponds to the light-receiving surface 68of the pixel array 38. In the observation plane, the diffraction patternof the diffracted wavefront 36 can be characterized by a spatiallyvarying intensity profile 70 along the grating axis 30 in thelight-receiving surface 68 of the pixel array 38. It is to be noted thatin FIG. 5, the grating axis 30 is parallel to the pixel axis 58.

The diffraction grating 28 and the pixel array 38 are disposed relativeto each other such that the light-receiving surface 68 of the pixelarray 38 is positioned in the near-field diffraction region, or simplythe near field, of the diffraction grating 28, as described inco-assigned international PCT patent application No. PCT/CA2017/050686.In the near-field diffraction regime, the Fresnel diffraction theory canbe used to calculate the diffraction pattern of waves passing through adiffraction grating. Unlike the far-field Fraunhofer diffraction theory,Fresnel diffraction accounts for the wavefront curvature, which allowscalculation of the relative phase of interfering waves. To detect thediffracted wavefront 36 in the near field, the present techniques caninvolve maintaining a sufficiently small separation distance 72 betweenthe top surface 48 of the diffraction grating 28, where refractive indexmodulation pattern 32 is formed and diffraction occurs, and thelight-receiving surface 68 of the underlying pixel array 38, where thediffracted wavefront 36 is detected. In some implementations, this caninvolve selecting the separation distance 72 to be less than about tentimes a center wavelength of the optical wavefront 26.

In the near-field diffraction regime, the intensity profile 70 of thediffracted wavefront 36 produced by a periodic diffraction grating 28generally has a spatial period 74 that substantially matches the gratingperiod 34 of the diffraction grating 28 as well as a shape thatsubstantially matches the refractive index modulation pattern 32 of thediffraction grating 28. For example, in FIG. 5, the diffraction patternof the diffracted wavefront 36 detected by the light-sensitive pixels 40of the pixel array 38 has a square-wave, or two-step, intensity profile70 that substantially matches that of the refractive index modulationpattern 32 of the binary phase diffraction grating 28. In the presentdescription, the term “match” and derivatives thereof should beunderstood to encompass not only an “exact” or “perfect” match betweenthe intensity profile 70 of the detected diffracted wavefront 36 and theperiodic refractive index modulation pattern 32 of the diffractiongrating 28, but also a “substantial”, “approximate” or “subjective”match. The term “match” is therefore intended to refer herein to acondition in which two features are either the same or within somepredetermined tolerance of each other. Another feature of near-fielddiffraction by a periodic diffraction grating is that upon varying theangle of incidence 76 of the incoming optical wavefront 26 on thediffraction grating 28, the intensity profile 70 of the diffractedwavefront 36 is laterally shifted along the grating axis 30, butsubstantially retains its period 74 and shape, as can be seen from thecomparison between solid and dashed wavefront lines in FIG. 5.

In the embodiment illustrated in FIG. 5, the color filter array 42 has aBayer pattern, of which FIG. 5 depicts a row of alternating green (G)and blue (B) filters. FIG. 5 also depicts schematically the propagationof light through the device 20. In operation, the diffraction grating 28receives and diffracts an optical wavefront 26 originating from thescene 22 to generate a diffracted optical wavefront 36, as mentionedabove. The color filter array 42 receives and spatio-spectrally filtersthe diffracted optical wavefront 36 prior to its detection by theunderlying pixel array 38. The operation of the light field capturedevice 20 is therefore based on a directly spatio-and-chromaticallysampled diffracted wavefront 36 enabled by the provision of a periodicdiffraction grating 28 deposed on top of a sensor structure including acolor filter array 42 and an underlying pixel array 38.

Upon being optically coupled to an underlying pixel array 38, thediffraction grating 28 convolves light phase information with a standard2D image, so that the intensity profile 70 of the diffraction pattern ofthe detected diffracted wavefront 36 can generally be written as amodulated function I˜I_(mod)(depth info)×I_(base) (2D image) including amodulating component I_(mod) and a base component I_(base). The basecomponent I_(base) represents the non-phase-dependent optical wavefrontthat would be detected by the pixel array 38 if there were nodiffraction grating 28 in front of it. In other words, detecting thebase component I_(base) alone would allow a conventional 2D image of thescene 22 to be obtained. Meanwhile, the modulating component I_(mod),which is generally small compared to the base component I_(base) (e.g.,ratio of I_(mod) to I_(base) ranging from about 0.1 to about 0.3), is adirect result of the phase of the incident optical wavefront 26, so thatany edge or slight difference in incidence angle will manifest itself asa periodic electrical response spatially sampled across the pixel array38. The sensitivity to incidence angle, and therefore the angularresolution, depends on the specific design of the diffraction grating28.

It will be understood that the intensity profile 70 of the diffractedwavefront 36 that is detected by the pixel array 38 afterspatio-spectral filtering by the color filter array 42 is a combinationor superposition of the portions of the diffracted wavefront 36 filteredby the red filters, the portions of the diffracted wavefront 36 filteredby the green filters, and the portions of the diffracted wavefront 36filtered by the blue filters. As such, using a standard RGB Bayerpattern as an example, the modulating component I_(mod) and the basecomponent I_(base) of the intensity profile I can be split into theirrespective color components as follows:I _(R) ˜I _(mod,R)(depth info)×I _(base,R)(2D image),  (3)I _(G) ˜I _(mod,G)(depth info)×I _(base,G)(2D image),  (4)I _(B) ˜I _(mod,B)(depth info)×I _(base,B)(2D image).  (5)

In FIG. 5, the intensity profiles I_(G) and I_(B) are depicted in dashedand dotted lines, respectively. In some implementations, theseindividual color channels may be treated independently to createindividual color channel light field scenes. These individual scenes maybe compared to provide both another cue for depth and scene analysis aswell as image compression, since the image may be transferred inindividual color channels each with its own respective file.

Referring still to FIG. 5, as mentioned above, in some embodiments thepixel array 38 has a pixel pitch 62 along the grating axis 30 that issmaller than the grating period 34 of the diffraction grating 28. Thismeans that when the light-receiving surface 68 of the pixel array 38 isin the near field of the diffracting grating 28, the pixel pitch 62 ofthe pixel array 38 along the grating axis 30 is also smaller than thespatial period 74 of the intensity profile 70 along the grating axis 30of the detected diffracted wavefront 36. When this condition isfulfilled, a complete period of the intensity profile 70 of the detecteddiffracted wavefront 36 will be sampled by at least two adjacent pixelbanks of the pixel array 38, each of these pixel banks sampling adifferent spatial part of the intensity profile 70 over a full cycle. Inthe present description, the term “pixel bank” refers to a group oflight-sensitive pixels of the pixel array that are arranged along a linewhich is perpendicular to the grating axis of the overlying diffractiongrating. That is, two adjacent pixel banks are separated from each otherby a distance corresponding to the pixel pitch along the grating axis.For example, in FIG. 5, each pixel bank of the pixel array 38 extendsperpendicularly to the plane of the page.

Depending on the application, the ratio R of the grating period 34 ofthe diffraction grating 28 to the pixel pitch 62 of the pixel array 38along the grating axis 30 can take several values. In someimplementations, the ratio R can be equal to or greater than two (i.e.,R≥2); or equal to a positive integer greater than one (i.e., R=(n+1),where n={1, 2, . . . }); or equal to an integer power of two (i.e.,R=2n, where n={1, 2, . . . }); or the like. In some implementations, itmay be beneficial or required that the grating period 34 be not onlylarger than, but also not too close to the pixel pitch 62 along thegrating axis 30. For example, in some implementations, it may beadvantageous that the grating period 34 be at least about twice theunderlying pixel bank pitch 62 to allow for each pair of adjacent pixelbanks to sufficiently subsample the resultant modulated diffractedwavefront 36, whose spatial modulation rate is dictated by theproperties of the diffraction grating 28, near or at Nyquist rate. ThisNyquist, or nearly Nyquist, subsampling can allow for the direct removalof the modulating component I_(mod) from the measured signal I bystandard signal processing techniques. Once removed, the modulatingsignal I_(mod) may be manipulated independently of the base componentI_(base).

For example, in FIG. 5, the ratio R of the grating period 34 to thepixel pitch 62 along the grating axis 30 is substantially equal to two.It will be understood that in such a case, adjacent pixel banks willsample complimentary spatial phases of the intensity profile 70 of thedetected diffracted wavefront 36, that is, spatial parts of theintensity profile 70 that are phase-shifted by 180° relative to eachother. This can be expressed mathematically as follows:|ϕ_(bank,n+1)−ϕ_(bank,n)|=π, where ϕ_(bank,n+1) and ϕ_(bank,n) are thespatial phases of the intensity profile 70 measured by the (n+1)^(th)and the n^(th) pixel banks of the pixel array 38, respectively. Such aconfiguration can allow for a direct deconvolution of the modulatingcomponent I_(mod) and the base component I_(base) through thesubsampling of the interference pattern resulting from the incident wavefronts interaction:I _(base)=½[I(bank_(n))+I(bank_(n+1))],  (6)I _(mod)=½[I(bank_(n))−I(bank_(n+1))].  (7)

It is to be noted that I(bank_(n))±I(bank_(n+1)) in Equations (6) and(7) are generally obtained by summing the intensities measured by thepixels of all the rows in the associated pixel bank.

Referring still to FIG. 5, the diffraction grating 28 has a duty cycleof 50% (i.e., ridges 52 and grooves 54 of equal width), and eachlight-sensitive pixel 40 is positioned under and in vertical alignmentwith either a corresponding one of the ridges 52 or a corresponding oneof the grooves 54. However, other arrangements can be used in otherapplications.

In a standard RGB Bayer pattern, because the red and blue filters arealways located in adjacent pixel banks, the signals I_(R) and I_(B),which are associated with the sparsely sampled red and blue components,will be in antiphase relative to each other. Meanwhile, because greenfilters are present in all pixel banks, the signal I_(G), which isassociated with the densely sampled green components, will contain bothin-phase and out-of-phase contributions.

In FIG. 5, the diffraction grating 28 is oriented with respect to theunderlying pixel array 38 so that the grating axis 30 is parallel to oneof the two orthogonal pixel axes 58, 60. Referring to FIGS. 6A and 6B,in other applications the grating axis 30 can instead be oblique to theorthogonal pixel axes 58, 60. It is to be noted that in suchconfigurations, the pixel pitch 62 along the grating axis 30 may remainsmaller than the grating period. It is also to be noted that pixel bankssuch as defined above, that is, groups of pixels arranged along a linetransverse to the grating axis 30 of the overlying diffraction grating28 can also be defined in oblique configurations. For example, FIG. 6Aincludes a first group of pixels 40 ₁ that belong to a first pixel banklocated under ridge 52, and a second group of pixels 40 ₂ that belongsto a second pixel bank located at an adjacent groove 54.

In the examples described so far, the diffraction grating assemblyincluded a single diffracting grating. However, referring to FIG. 7, inother cases, the diffraction grating assembly 24 includes a plurality ofdiffracting gratings 28 a, 28 b, where the diffracting gratings 28 a, 28b are arranged in a two-dimensional grating array disposed over thecolor filter array 42. In FIG. 7, the diffracting grating assembly 24includes sixteen diffraction gratings, but this number can be varied inother applications. For example, depending on the application, thenumber of diffraction gratings 28 a, 28 b in the diffraction gratingassembly 24 can range from one to up to millions (e.g., a 20-megapixelpixel array 38 could have up to 2.8 million diffraction gratings on topof it). It is to be noted that other than their grating axisorientation, the diffraction gratings 28 in FIG. 7 are identical,although this is not a requirement of the present techniques.

In FIG. 7, the diffraction gratings 28 include a first set 80 a ofdiffraction gratings 28 and a second set 80 b of diffraction gratings28, the grating axes 30 a of the diffraction gratings 28 of the firstset 80 a extending substantially perpendicularly to the grating axes 30b of the diffraction gratings 28 of the second set 80 b. In FIG. 7, thediffraction gratings 28 of the first set 80 a and second set 80 b arearranged to alternate in both rows and columns, resulting in acheckerboard pattern. Of course, any other suitable regular or irregulararrangement, pattern or mosaic of orthogonally oriented gratings can beenvisioned in other applications.

In some applications, the light field capture device can includewavefront conditioning optics in front of the diffraction grating. Thewavefront conditioning optics can be configured to collect, direct,transmit, reflect, refract, disperse, diffract, collimate, focus orotherwise act on the optical wavefront incident from the scene prior toit reaching the diffraction grating assembly. The wavefront conditioningoptics can include lenses, mirrors, filters, optical fibers, and anyother suitable reflective, refractive and/or diffractive opticalcomponents, and the like. In some implementations, the wavefrontconditioning optics can include focusing optics positioned andconfigured to modify the incident wavefront in such a manner that it maybe sampled by the light field capture device.

Referring now to FIG. 8, another possible example of a light fieldcapture device 20 is illustrated and includes dispersive optics 84disposed in a light path of the optical wavefront 26 between the sceneand the diffraction grating assembly 24. The dispersive optics 84 isconfigured to receive and disperse the incoming optical wavefront 26.The dispersive optics 84 can be embodied by any optical component orcombination of optical components in which electromagnetic beams aresubject to spatial spreading as a function of wavelength as they passtherethrough (e.g., by chromatic aberration). In FIG. 8, the dispersiveoptics 84 is a focusing lens, for simplicity. However, it will beunderstood that, in other embodiments, the dispersive optics 84 can beprovided as an optical stack including a larger number of opticalcomponents (e.g., focusing and defocusing optics) that together act todisperse the optical wavefront 26 before it impinges on the diffractiongrating assembly 24 (e.g., due to their intrinsic chromatic aberration).

For exemplary purposes, it is assumed in FIG. 8 that the opticalwavefront 26 originating from the scene 22 is a superposition of wavescontaining multiple wavelengths of light, for example a green component(dashed line) and a blue component (dotted line). Each color componentof the optical wavefront 26, by the nature of its energy-dependentinteraction with the dispersive optics 84, will follow a slightlydifferent optical path, leading to a chromatic dependence in thephase-shift introduced by the diffraction grating 28. In other words,the chromatic spread of the optical wavefront 26, as sampled through theangle-dependent diffraction produced by the diffractive grating 28, canbe taken advantage of to provide coarse depth information about theoptical wavefront 26. In such scenarios, as described below, the finerdetails of the depth information can be obtained from a comparativeanalysis of the modulating components I_(mod,R) and I_(mod,B), which arephase-shifted relative to each other due to their optical pathdifferences, as sampled by the color filter array 42. That is, becausered and blue lights are focused differently by the dispersive optics 84because of chromatic aberration, this difference can provide finerdetail for depth acquisition.

Referring to FIG. 9, there is shown a flow diagram of an embodiment of amethod 200 for processing light field image data corresponding to alight field from a scene. The light field image data could be obtainedwith a light field capture device 20 such as those described above(e.g., the one in FIG. 8), or with another light field capture device,for example through a rolling or global shutter-like process.

The image data captured by the light field capture device 20 of FIG. 10represents a discretized sampling of a continuous light field orwavefront 26 incident from an observable scene 22. This discretizationcan occur through various optical processes and interactions: thefield-of-view limitations due to the finite aperture size of the device20, the limitations imposed by lenses in terms of what information canbe focused or transmitted, the limitations in terms of sub-samplingresolution due to the size of the pixels, etc. In the device 20 of FIG.7, the fully continuous light field 26 in (X, Y, Z) is projected to adiscretized (X, Y) plane and subsampled for color and spatial intensityby the pixels 40 of the pixel array 38. A phase component (Z) can alsobe measured due to the provision of the diffraction grating assembly 24.This phase measurement is also a discretized subsampling of the fulllight field 26, this discretization being controlled by the gratingdesign and arrangement of the diffraction grating assembly 24. Thedevice 20 of FIG. 7 aims to retain some information lost through thediscretization process inherently involved in imaging. Moreparticularly, the phase-based nature of the diffraction grating assembly24 is used to create a known perturbation on the phase of the wavefrontwhich would be otherwise unmeasurable, and which happens to be angledependent. This adds an additional and independent variable which can beused to describe the measured light field image data.

Referring still to FIG. 7, light 26 incident on the image capture device20 may first encounter a macroscopic optical element (not shown in FIG.7—FIG. 8, in which the device includes focusing optics disposed in frontof the diffraction grating assembly) that acts to collect and focusincoming light 26. This optical element could be a lens, eithertransmissive or reflective, a mirror, or a fiber-optic cable. Thisoptical element may also filter the light spectrally for the visiblewavelengths by excluding infrared and ultraviolet light, or, thesespectral regions may be coarsely filtered at another point in theoptical stack by a separate macroscopic filter.

The ability of this optical element to focus incident light 26 isrelated to the light's origin distance from the focal plane of theoptical element and the focusing power of the optical element. Whenincident light emanates from focal or hyperfocal distances, or opticalinfinity, its wavefront will, in general, effectively be tightly focusedonto very few pixels 40. Light that is insufficiently focused, orfocused too strongly, will continue to spread across the pixel array 38,leading to a curved wavefront which can expand over multiple pixels 40.

The focused light then encounters the diffraction grating assembly 24disposed atop the imaging system (i.e., color filter array 42 and pixelarray 38). In FIG. 7, the diffraction grating assembly 24 is atransmissive diffracting structure made up of two sets of orthogonallyoriented diffraction gratings 28 a 28 b arranged to alternate in bothrows and columns (checkerboard pattern). The diffraction gratingassembly 24 is used to create a diffracted wavefront 36 having anangularly dependent diffractive pattern related to the curvature andrate of spread of the wavefront formed after the focusing opticalelement.

The diffracted wavefront 36 is further spectrally filtered by theunderlying color filter array 40 into two or more discrete spectralcomponents. The intensity of this spectrally filtered, diffractedwavefront is finally measured by the underlying pixels 40 and convertedinto electrical signals representative of the intensity of lightcaptured by the pixels 40. The electrical signals can then be digitizedinto “raw” light field image data, typically arranged or formatted intoan image frame of rows and columns of data pixels. This light fieldimage data will generally be a collection of intensity, spectral, andangular information about the original wavefront, sampled at the pixellevel, which can be summarized as follows: I=Angular Information(wavelength, diffraction grating assembly location)+2D (wavelength,pixel location). The present description discloses image processingtechniques in which this light field image data is processed toreconstruct a light field image of the observed scene 22. Depending onthe application, the present techniques can provide a full resolution 2Dimage, a 3D point cloud, a depth map/image and/or a linearlyreconstructed light field image from the captured light field imagedata.

Returning to FIG. 9, in some implementations, the image processingmethod 200 can include a step 202 of parsing the pixel data according toa primary and at least one secondary colors into corresponding primaryand secondary color channels. The splitting or parsing of the raw lightfield image data into a plurality of individual spectral or colorchannels forms individual sub-images. This step 202 generally occurs atthe hardware readout level. The individual pixels are read, and thecolor filter mosaic is known. For example, for a 12-megapixel camerausing standard Bayer patterning there would be a 6-megapixel green colorchannel, a 3-megapixel blue channel and a 3-megapixel red channel.

As mentioned above, the raw light field image data in each color channelcan be expressed as a modulated function I˜I_(mod)(depthinfo)×I_(base)(2D image) including a modulating component I_(mod) and abase component I_(base). The base component I_(base) represents thenon-phase-dependent optical wavefront from which a conventional 2D imageof a scene can be obtained. The modulating component I_(mod) resultsfrom the phase-dependent perturbation to the incident wavefront createdby the diffraction grating assembly 24. Therefore, in the case of astandard RGB Bayer pattern, the intensity profile of each color channelcan be written as follows I_(N)˜I_(mod,N)(depth info)×I_(base,N)(2Dimage), where N=color or spectral channel, for example red (R), green(G) and blue (B)—see Equations (3) to (5). As mentioned above, thepresent techniques are not limited to Bayer-type patterns, but can beapplied to any appropriate mosaic color pattern.

In applications where the diffraction grating assembly 24 includesmultiple sets of diffraction gratings 28 a, 28 b (e.g., having differentorientations, as in FIG. 7, different grating periods, or otherdifferent properties), the method 200 of FIG. 9 can include a step ofparsing or splitting the spectral channels in accordance with thepattern of the 2D grating array into which the multiple sets ofdiffraction gratings 28 a, 28 b are arranged. For example, in FIG. 7,each one of the parsed red, green and blue color channels will be parsedonce more into a first sub-channel associated with the first set ofdiffraction gratings 28 a and a second sub-channel associated with thesecond set of diffraction gratings 28 b. This step ensures that the datapixels of vertically oriented gratings 28 a will be separated andtreated independently from horizontally oriented gratings 28 b. This canallow the responses of vertically and horizontally oriented gratings tobe treated independently from one another, which can enhance processingspeed.

In some implementations of step 202, the color channels into which thelight field image data is parsed includes a main color channel and oneor more secondary or auxiliary color channels. In the presentdescription, a main color channel is a color channel having pixels inall pixel banks, such that pixels of the main color channel in adjacentpixel banks sample different spatial parts of the intensity profile ofthe diffracted wavefront 36 over a full cycle. As mentioned above, apixel bank refers herein to a group of pixels 40 of the pixel array 38that are arranged along a line which is perpendicular to the gratingaxis 30 a, 30 b of the overlying diffraction grating 28 a, 28 b. In someimplementations, there could be two or more main channels if there aremore than one color having pixels in all pixel banks.

In the device 20 of FIG. 7, the green channel is a main color channelsince the green pixels are present in all pixel banks. This means thatthe green pixels in the n^(th) pixel bank will sample a spatial phase ofthe intensity profile of the diffracted wavefront 36 which isphase-shifted by 180° relative to the spatial phase sampled by the greenpixels in the (n±1)^(th) pixel bank. As a result, the signal I_(G)associated with the more densely sampled green pixels contains bothin-phase and out-of-phase contributions. In other words, complimentaryphases of the intensity profile of the diffracted wavefront 36 will besampled by green pixels in alternating pixel banks. Meanwhile, the redand blue channels are secondary color channels since all the red pixelssample a same first spatial phase of the intensity profile of thediffracted wavefront 36 and all the blue pixels sample a same secondspatial phase of the intensity profile of the diffracted wavefront 36,the first and second spatial phases being phase-shifted by 180° relativeto each other. This means that the signals I_(R) and I_(B) in adjacentpixel banks are in antiphase relative to each other. Of course, othermosaic color patterns may use different color channels as main andsecondary channels.

Referring still to FIG. 9, some implementations of the method 200 canalso include a step 204 of using the main color channel to provide orextract a coarse raw base component, from which the local influence ofthe modulation imparted to the signal by the diffraction gratingassembly 24 is removed.

In the example of FIG. 7, the main color channel is the more denselysampled green channel, which can sample both the major phase componentsfrom which the modulation component of the underlying diffractionpattern can be canceled. Mathematically, this step can involveextracting I_(base,G) from I_(G)˜I_(mod,G)×I_(base,G) by cancelingI_(mod,G) using the following equation:I _(base,G)=½[I _(G)(bank_(n))+I _(G)(bank_(n+1))].  (8)

Alternatively, the modulating component I_(mod,G) may be removed byFourier filtering.

The method 200 can also include a step 206 of using the main colorchannel to provide or extract a coarse raw modulating component, whichwill be used to obtain coarse phase/depth information about the scene22.

Again, in the example of FIG. 7, the main color channel is the moredensely sampled green channel. Mathematically, this step can involveextracting I_(mod,G) from I_(G)˜I_(mod,G)×I_(base,G) by cancelingI_(base,G) using the following equation:I _(mod,G)=½[I _(G)(bank_(n))−I _(G)(bank_(n+1))].  (9)

The modulating component I_(mod,G) can provide coarse angular or phaseinformation about the incident optical wavefront, from which light fieldimages and/or depth maps can be generated. As described above, themodulating component I_(mod,G) represents the wavefront's phasedifference pixel by pixel, so that I_(mod,G) contains information aboutthe phase or spread of the incident optical wavefront 26 from the scene22. More particularly, the modulating component I_(mod,G) is the resultof the tilt/angular spread of the green component of the incidentwavefront 26 (e.g., caused by focusing optics in front of thediffraction grating 24 assembly—see FIG. 8) and the phase offset causedby the diffraction grating assembly 24.

Referring still to FIG. 9, the method 200 can further include a step 208of determining the raw base components I_(base,R) and I_(base,B) and theraw modulating components I_(mod,R) and I_(mod,B) of the secondary blueand red channels can be obtained from I_(R)˜I_(mod,R)×I_(base,R) andI_(B)˜I_(mod,B)×I_(base,B), respectively, using I_(base,G) andI_(mod,G).

In some implementations, the difference between I_(G)(bank_(n)) andI_(G)(bank_(n+1)) can give the sign of the modulation pattern impartedby the diffraction grating. That is, ifI_(G)(bank_(n))<I_(G)(bank_(n+1)), then it can be concluded thatbank_(n) undergoes destructive interference and bank_(n+1) undergoesconstructive interference, and vice versa ifI_(G)(bank_(n))>I_(G)(bank_(n+1)). Therefore, in a scenario whereI_(G)(bank_(n))<I_(G)(bank_(n+1)), with red pixels in bank_(n) and bluepixels in bank_(n+1), it can be deduced that I_(R)(bank_(n))<I_(base,R)and that I_(B)(bank_(n+1))>I_(base,B), such thatI_(base,R)=I_(R)(bank_(n))+I_(mod,R) andI_(base,B)=I_(B)(bank_(n+1))−I_(mod,B). The knowledge of the sign of[I_(G)(bank_(n))−I_(G)(bank_(n+1))] can therefore allow to determinewhether to apply a constructive or destructive phase offset toI_(R)(bank_(n)) and I_(B)(bank_(n+1)) to obtain (I_(base,R), I_(mod,R))and (I_(base,B,) and I_(mod,B)).

To obtain the amplitude of the phase offset for red light, that is,I_(mod,R) one can use a relationship between the wavefront spread ofgreen light and the wavefront spread of red light in the image capturedevice (e.g., caused by the diffraction grating and focusing optics) toderive I_(mod,R) from I_(mod,G). Similarly, to obtain the amplitude ofthe phase offset for blue light, that is, I_(mod,B), one can use arelationship between the wavefront spread of green light and thewavefront spread of blue light in the image capture device (e.g., causedby the diffraction grating and focusing optics) to derive I_(mod,B) fromI_(mod,G). In some implementations, fine angular information about thewavefront spread can be obtained from the phase-shifted red and bluesecondary modulating components I_(mod,R) and I_(mod,B), as describedbelow. Finally, the red and blue base components can be obtained usingI_(base,R)=I_(R)(bank_(n))+I_(mod,R) andI_(base,B)=I_(B)(bank_(n+1))−I_(mod).

Referring still to FIG. 9, the method can include a step 210 of usingthe raw base components I_(base,G), I_(base,R) and I_(base,B) toreconstruct a full color 2D image of the scene 22 in similar fashionsused by traditional demosaicing algorithms, for example by using thegreen base component I_(base,G) as a normalizing basis set forI_(base,R) and I_(base,B). This full color 2D image represents thephase-independent raw image that would be captured by the pixel array 38in the absence of the diffraction grating assembly 24 in front of it.This generation of the 2D image can be performed for one or more of thefollowing non-limiting reasons: to separate depth information from 2Dinformation for compression purposes; to be able to provide the imagesignal processor (ISP) with a 2D image, as is expected for displaypurposes; and the 2D image is one of the light field components.

Referring still to FIG. 9, in some implementations, the method 200 caninclude a step 212 of creating a depth map of the scene from themodulating component I_(mod,G) of the main color channel, and optionallyfrom the modulating components I_(mod,R) and I_(mod,B) of the secondarycolor channels. This step main also involve at least one color radialtransfer function calibrated to provide object distance information fromthe modulating component of an associated one of the color channels.

In some implementations, the color radial transfer functions (CRTFs)relate the diffraction grating pattern of a diffraction grating as afunction of angle to the distance of an object is away from the camera.In other words, the CRTFs may allow one to quantify the modulationimparted for two phase components of a diffraction grating, phase0 andphase180, without apriori knowledge of the scene.

For the purpose of explanation only and without being limitative to thescope of the present description, let there be considered a situation inwhich a camera is capturing images of a white light point-source inpredetermined intervals at increasing distances away from the camera, asshown in FIG. 10. In the case of a camera without a diffraction grating,the sensor would spatio-chromatically sample the overall intensity ofthe white light as a function of the underlying Bayer pattern, alsoshown in FIG. 10. This spatio-chromatic sampling of the white lightsource leads to a voltage measured by each pixel roughly proportional tothe spectral intensity of the white light filtered by the spectrallydependent transmission of a given color filter array component:Pixel_(Red)∝Intensity(λ)×Transmission(λ)  (10.1)or Pixel_(Red)=2D(Red)  (10.2)

The TDM's modulated phase component will also vary from pixel bank topixel bank:Pixel_(n)∝Intensity(λ)×Transmission(λ)×Modulation_(Phase0)(λ,Z)  (11.1)Pixel_(n+1)∝Intensity(λ)×Transmission(λ)×Modulation_(Pphase180)(λ,Z)  (11.2)

In the case of a Bayer patterned sensor where the underlying colorfilters spatio-chromatically sample the light at a rate greater than theperturbation frequency caused by the diffraction grating (i.e. the pixelarray having a pixel pitch along the grating axis that is smaller thanthe grating period), the three-color channels, as shown in FIG. 10,become:Pixel_(Red)=2D(Red)×Modulation_(Phase0)(λ,Z)  (12.1)Pixel_(Green1)=2D(Green)×Modulation_(Phase180)(λ,Z)  (12.2)Pixel_(Green2)=2D(Green)×Modulation_(Phase0)(λ,Z)  (12.3)Pixel_(Blue)=2D(Blue)×Modulation_(Phase180)(λ,Z)  (12.4)

Or, using the formalism developed above:Pixel_(Red) I _(BaseR)(Z)×I _(ModR) _(Phase0) (λ,Z)  (13.1)Pixel_(Green1) =I _(BaseG1)(Z)×I _(ModR) _(Phase180) (λ,Z)  (13.2)Pixel_(Green2) =I _(BaseG2)(Z)×I _(ModR) _(Phase0) (λ,Z)  (13.3)Pixel_(Blue) =I _(BaseB)(Z)×I _(ModR) _(Phase180) (λ,Z)  (13.4)

The functional forms of equations 12.1 through 12.4 and 13.1 through13.4 are shown in FIG. 11A to 11D, in which full lines represent thecamera described in FIG. 10 without a diffraction grating and the dashedlines represent the same camera with a diffraction grating.

In the case of a Bayer patterned sensor, one can remove the influence ofthe diffraction grating on the image, or in other words, extract depthinformation, by having one color channel sample both phase components:Pixel_(n)=Intensity(n)×phase0(Z)  (14.1)Pixel_(n+1)=Intensity(n+1)×phase180(Z)  (14.2)

This brings the definition of a simplified main (primary) color channelradial transfer function whose functional form is shown in FIGS. 12A and12B:Δa(Z)=Pixel_(n)−Pixel_(n+1)=Δphase(Z)  (15)

The chromatic radial transfer function uses the difference between thetwo main color channel modulated components to remove intensity from theimage, allowing for a normalized value to relate the other two colorchannels who do not have both phase components and whose intensities aretherefore variable with both the 2D image as well as the diffractiongrating modulation, or depth information.

A similar experiment to that imagined in FIG. 10 may be performed for asecondary color channel. In this case, one can use a known uniformintensity of light to measure the secondary color channel modulation asa function of angle or distance. This will provide the secondarychromatic radial transfer functions, as shown in FIG. 13.

Referring to FIG. 14, the primary radial chromatic transfer function,which works without apriori knowledge of the scene, may be used toacquire a normalized, scene independent, value to navigate the secondarychromatic radial transfer functions. Once the appropriate modulationvalue is known for the secondary color channels, the effect of thediffraction grating can be corrected for in the 2D image and thedifferences in the secondary and primary radial transfer functionsvalues, and their respective pixel values, can be compared to furtherenhance depth performance.

This concept can also be extended to the lens system and its achromaticaberrations and chromatically dependent focal differences through thesame processing method.

In some embodiments, the raw wavefront spread information contained inthe modulating component I_(mod,G) can be compared to an appropriategreen radial transfer function (GRTF) to yield a coarse depth map of thescene 22. More particularly, I_(mod,G) can provide relative phaseinformation and comparison with a calibrated, sensor-specific GRTF canallow one to obtain absolute phase information from the relative phaseinformation provided by I_(mod,G). In other words, this means that therelative phase information provided by I_(mod,G) can be mapped to awavefront spread using the GRTF, this wavefront spread itselfcorresponding to an object position relative to the focal plane.

Optionally, the coarse depth map obtained from I_(mod,G) and the GRTFcan be corrected or enhanced using data from the secondary colorchannels (e.g., I_(mod,R), I_(mod,B) and their associated red and blueradial transfer functions RRTFs and BRTFs). This means that the directcomparison of the modulating components I_(mod,G), I_(mod,R) andI_(mod,B) for the three color channels is done through measured CRTFsfor each color channel in order to establish object distance.

As mentioned above, each one of the CRTFs provides an empirical fit ofthe directly measured wavefront spread as a function of the focal planeof the image capture device 20. Collections of these sparse data-sets ofcalibration images captured at varying distances from the focal planeare fit as a function of their distance from the camera and provide adirect empirical measure of the radially extended spread of thewavefront as a function of distance from the focal plane as measured bythe modulating components I_(mod,G) (main), and I_(mod,R), and I_(mod,B)(secondary) of the image and sampled by the diffraction grating assembly24. These functions provide a method for directly mapping the measuredrelative phase/wavefront spread information given from I_(mod,G),I_(mod,R), and I_(mod,B) to an absolute object distance from the imagesensor, and therefore, depth (Z):Object(Z)˜RGB2D Image×CRTF(r,ϕ,θ,n,λ),  (10)where z is the distance from the detection plane of image capture deviceof a given object in the scene 22, RGB 2D Image is the raw 2D imagegiven by I_(base,G), I_(base,R), and I_(base,B), and the CRTF is afitted 5D function of the polar coordinates r, ϕ, θ from the focal planeof the image, the pixel number n (and therefore spatial location on thepixel array 38), and the incident wavelength λ. The CRTFs can be sampledin a single image for all spectral elements as sampled by thediffractive elements and imaging devices. That is, all CRTFs could becaptured at once using white light and be subsampled directed by a colorfilter array. These functions may be tailored through the design of thediffractive grating assembly 24. The CRTFs can provide a macroscopicdescription of the micro sampling of the diffraction pattern resultingfrom the arrangement of the diffraction gratings 28 of the diffractiongrating assembly 24. It is to be noted that in absolute depthimplementations, one could use a general RTF or use color-specific CRTFsto increase depth accuracy.

It will be readily understood that while the implementations describedabove apply to circumstances where the pixel sampling frequency isgreater than the period of the grating pattern, the processing methoddescribed herein may in some instance be useful in association withsimilar imaging devices for which the sampling frequency is equal to orless than the grating period (or in words, where the pixel array has apixel pitch along the grating axis that is the same or greater than thegrating period). In such a cases, the steps of reconstructing the 2Dimage may be omitted, as there is no chromatically-dependent blurpattern created in the 2D image by the action of the grating—both phasecomponents are already included in each grating. In such embodiments themicro-chromatic dependence of the signal and associated is lost, but themacro-chromatic dependence may still be useful to provide depthinformation. In other words, the provision of different color channelscombined with associated color radial transfer functions can providedepth information through comparisons of the different color channels toobtain their relative difference in focal position.

Of course, numerous modifications could be made to the embodimentsdescribed above without departing from the scope of the presentdescription.

The invention claimed is:
 1. A method of imaging a scene, the methodcomprising: diffracting an optical wavefront originating from the sceneto generate a diffracted optical wavefront; detecting the diffractedoptical wavefront in a near-field regime using a pixel array comprisinga plurality of light-sensitive pixels each associated with a colorfilter, thereby obtaining pixel data, the color filters forming a colormosaic defining a primary color and one or more secondary colorsarranged such that different neighboring pixels associated with theprimary color detect different spatial parts of the diffracted opticalwavefront over a full cycle of the diffracted optical wavefront; parsingthe pixel data according to the primary and secondary colors intocorresponding primary and secondary color channels; determining a mainbase component and a main modulating component of the diffracted opticalwavefront associated with the primary color channel; determining, basedon the main modulating component, a secondary base component of thediffracted optical wavefront associated with each secondary channel;reconstructing a 2D image of the scene based on the main and secondarybase components; and creating a depth map of the scene based on at leastone color radial transfer function calibrated to provide object distanceinformation from the modulating component of an associated one of thecolor channels.
 2. The method of claim 1, wherein the primary colorchannel comprises a green color channel, and the one or more secondarycolor channels comprise a red color channel and a blue color channel. 3.The method of claim 2, wherein the color mosaic is a Bayer pattern. 4.The method of claim 1, wherein determining the main base component andthe main modulating component comprises summing and subtracting pixeldata from neighboring pixel banks associated with the primary color,respectively.
 5. The method of claim 1, wherein determining thesecondary base component associated with each secondary color channelcomprises: determining a secondary modulating component associated withthe secondary color channel based on the main base component and themain modulating component; and determining the secondary base componentbased on the secondary modulating component.
 6. The method of claim 1,wherein creating the depth map comprises creating a coarse depth mapbased on the main modulating component and the associated color radialtransfer function.
 7. The method according to claim 6, wherein creatingthe coarse depth map comprises: determining relative depth informationfrom the main modulating component; and determining absolute depthinformation from a comparison of the relative depth information to thecolor radial transfer function associated with the primary colorchannel.
 8. The method of claim 6, wherein creating the depth mapcomprises adjusting the coarse depth map based on the one or moresecondary modulating components and the associated one or more colorradial transfer functions.
 9. The method of claim 1, further comprisingproviding the diffraction grating with a grating period that is greaterthan a pixel pitch of the pixel array.
 10. The method of claim 9,wherein the grating period is twice the pixel pitch.
 11. Anon-transitory computer readable storage medium having stored thereoncomputer executable instructions for generating three-dimensional imagedata of a scene from a diffracted optical wavefront originating from thescene and having been detected in a near-field regime using a pixelarray comprising a plurality of light-sensitive pixels each associatedwith a color filter, thereby obtaining pixel data, the color filtersforming a color mosaic defining a primary color and one or moresecondary colors arranged such that different neighboring pixelsassociated with the primary color detect different spatial parts of thediffracted optical wavefront over a full cycle of the diffracted opticalwavefront, the computer executable instructions, when executed by aprocessor having received the pixel data, cause the processor to performthe following steps: parsing the pixel data according to the primary andsecondary colors into corresponding primary and secondary colorchannels; determining a main base component and a main modulatingcomponent of the diffracted optical wavefront associated with theprimary color channel; determining, based on the main modulatingcomponent, a secondary base component of the diffracted wavefrontassociated with each secondary channel; reconstructing a 2D image of thescene based on the main and secondary base components; and creating adepth map of the scene based on at least one color radial transferfunction calibrated to provide object distance information from themodulating component of an associated one of the color channels.
 12. Thenon-transitory computer readable storage medium of claim 11, wherein theprimary color channel comprises a green color channel, and the one ormore secondary color channels comprise a red color channel and a bluecolor channel.
 13. The non-transitory computer readable storage mediumof claim 12, wherein the color mosaic is a Bayer pattern.
 14. Thenon-transitory computer readable storage medium of claim 11, whereindetermining the main base component and the main modulating componentcomprises summing and subtracting pixel data from neighboring pixelbanks associated with the primary color, respectively.
 15. Thenon-transitory computer readable storage medium of claim 11, whereindetermining the secondary base component associated with each secondarycolor channel comprises: determining a secondary modulating componentassociated with the secondary color channel based on the main basecomponent and the main modulating component; and determining thesecondary base component based on the secondary modulating component.16. The non-transitory computer readable storage medium of claim 11,wherein creating the depth map comprises creating a coarse depth mapbased on the main modulating component and the associated color radialtransfer function.
 17. The non-transitory computer readable storagemedium according to claim 16, wherein creating the coarse depth mapcomprises: determining relative depth information from the mainmodulating component; and determining absolute depth information from acomparison of the relative depth information to the color radialtransfer function associated with the primary color channel.
 18. Thenon-transitory computer readable storage medium of claim 16, whereincreating the depth map comprises adjusting the coarse depth map based onthe one or more secondary modulating components and the associated oneor more color radial transfer functions.
 19. The non-transitory computerreadable storage medium of claim 11, wherein the diffraction grating hasa grating period that is greater than a pixel pitch of the pixel array.20. The non-transitory computer readable storage medium of claim 19,wherein the grating period is twice the pixel pitch.
 21. A system forimaging a scene, the system comprising: a diffracting grating configuredto diffract an optical wavefront originating from the scene to generatea diffracted optical wavefront; a pixel array comprising a plurality oflight-sensitive pixels configured to detect the diffracted opticalwavefront in a near-field regime, thereby obtaining pixel data; a colorfilter array interposed between the diffraction grating and the pixelarray, the color filter array comprising a plurality of color filterseach associated with a corresponding one of the plurality oflight-sensitive pixels and forming a color mosaic defining a primarycolor and one or more secondary colors arranged such that differentneighboring pixels associated with the primary color detect differentspatial parts of the diffracted optical wavefront over a full cycle ofthe diffracted optical wavefront; and a processor configured to: receivethe pixel data from the pixel array; parse the pixel data according tothe primary and secondary colors into corresponding primary andsecondary color channels; determine a main base component and a mainmodulating component of the diffracted optical wavefront associated withthe primary color channel; determine, based on the main modulatingcomponent, a secondary base component of the diffracted opticalwavefront associated with each secondary channel; reconstruct a 2D imageof the scene based on the main and secondary base components; and createa depth map of the scene using at least one color radial transferfunction calibrated to provide object distance information from themodulating component of an associated one of the color channels.
 22. Thesystem of claim 21, wherein the diffraction grating comprises atransmissive binary phase grating.
 23. The system of claim 21, whereinthe diffraction grating has a grating period, and wherein the pixelarray has a pixel pitch that is smaller than grating period.
 24. Thesystem of claim 21, wherein the color filters comprises red filters,green filters, and blue filters, wherein the color mosaic is a Bayerpattern, and wherein the primary color channel comprises a greenchannel, and the one or more secondary color channels comprise a redchannel and a blue channel.
 25. The system of claim 21, wherein theprocessor is configured to determine the main base component and themain modulating component by summing and subtracting pixel data fromneighboring pixel banks associated with the primary color, respectively.